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ABSTRACT 

This paper focuses on the lack of productive 
knowledge-building in social education research by raising issues 
related to the place of scholarship in the field, the nature of its 
goals, and whether empirical-scientific research can be carried out 
to meet those goals. Attention is drawn to the limits of educational 
research based on untenable assumptions about human behavior so that 
more fruitful perspectives on the role of research may be developed 
and so that future research undertakings may be more likely to 
produce knowledge to inform and sensitize decision-makers. The paper 
also examines the following topics: (1) teachers 1 lack of access to 
research findings and the limited use of such findings; (2) the 
contradiction between two goals -of research: the desire to help 
practitioners and the desire to contribute to the body of scientific 
knowledge; (3) the appropriateness of the empirical-scientific 
approach in building knowledge; and (4) how meta-analysis fits into 
the research picture. (KWL) 
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SCHOLARSHIP IN SOCIAL EDUCATION: 
ISSUES FROM AN EMPIRICAL, META-ANALYSIS RESEARCH PERSPECTIVE 



James P. Shaver 
Utah State University 

As egocentric or intimidating as it may sound, we are part of the 
intellectual history of educational research and, in particular, of social 
education research, a field within which it is common to lament the lack of 
productive efforts at knowledge-building. P<^~h^ps the most difficult thing 
that can be asked of anyone who is part of a Zeitgeist (as we are, in large 
part in social education, caught up in the dominant e -i^L: ical-scientific 
spirit of educational research) is that they disregard their frames of 
reference, the specific commitments bred by their own past actions, and 
examine critically and thoughtfully, without defensiveness, the endeavors in 
which they are engaged. Our consideration of scholarship involving 
"scientific-empirical" research, including meta-analysis, in social education 
will be aided if we can adopt such a perspective. 

I would like to pose some issues — some new, but most recurring, and 
unresolved, ones — in regard to our endeavors in that field. I could begin by 
spending considerable time in drawing fine definitional distinctions. 
However, I think there is sufficient general agreement in usage of the term 
"empirical research" or "scientific-empirical research" for the purposes of 
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our discussion today. Stated briefly, it is the traditional type of social 
science research, with a positivist orientation, through which we have 
attempted to build knowledge, using largely quantitative assessments and 
methods assumed to be realistically (that is, validly) related to the "real 
world" — that is, to an orderly reality which is there to be discovered. (For 
a close approximation to this usage, see the definitions of "empirical- 
analytic research" by Popkewitz & Tabachnick, 1982/. p. v, and Popkewitz, 
1986, pp. 14-17.) 

Scholarship By and For Whom? 

My focns today will be on the frequently lamented lack of productive 
knowledge-building in our field. A mundane, but crucial,, starting point is 
to note that of those college and university people who might be engaging in 
scholarship, and in particular empirical research in social education, very 
few are so involved. There is not *n overwhelming body of scholarship, 
quantitatively speaking. Theory and Research in Social Education ( TRSE ) is 
not inundated vith manuscripts; relatively few studies are published there in 
any one year, and very few studies in social education are published, 
elsewhere. 

Ours is basically a service profession, concerned with the education of 
young people and, for that reason, with the education of teachers. That 
orientation, while not necessarily contradictory to research interests, does 
take up resources of time and energy, and demands attention to a different 
set of concerns and questions. Viewed from that perspective, it is not 
surprising that research scholarship has not exactly proliferated in social 
education. 



For Whom or What Purpose is Research Done?? 

Given the research that is done, for whom is it done or for what 
purpose? Some argue that they do research to affect practice. That is, 
their purpose is to provide knowledge that teachers, curriculum specialists 
and supervisors, and college methods professors can use in making curriculum 
and instructional decisions. For those people, it must be disconcerting that 
social studies teachers seem rarely to be aware of research findings, much 
less users of them (Shaver, Davis, & Helburn, 1980). But that should not be 
surprising. If asked ourselves how often we use research findings in making 
instructional decisions in our own courses, I would suspect that most of us 
would have to answer as did those in an informal survey of Stanford 
University faculty by Eisner (1983) and of Utah State University faculty by 
me (1982b): Despite their admonitions to public school people to base 
practice on research, education faculty rarely do so themselves. 

There are, of course, a number of reasons why teachers pay so little 
attention to research, and find it of little help when they do. In one 
recent paper (Shaver, 1982b), I listed some fifteen possible reasons. Aside 
from teachers 1 lack of access to research findings, the limitations on use 
can be grouped in three clusters: (1) the research typically does not 
address issues or questions of interest to teachers; (2) research is 
typically not set in a context that is deemed valid by practitioners (for 
example, treatments tend to be of very short duration and carried out by 
people who are only temporarily in the schools); and, (3) research studies 
tend to produce inconsistent, evon contradictory, results. All told, neither 
the relevance nor the implications for curriculum or instruction are clear. 



On the other hand/ some researchers would argue that their intent is to 
build knowledge— some say, theory* — about social education. For those who 
conduct research from a theory-building orientation, the major disconcerting 
reality must be that despite some impressive, suggestive studies, Generally 
there is a lack of cumulative knowledge or empirically-based scientific 
theory in social education. 

Perhaps part of the problem is that researchers often claim both goals — 
that is, their intent is to help practitioners and to build theory (see, 
e.g., Cornbleth, 1986, p. 7).** Thomas Kuhn. (1970, pp. 19, 164) pointed out 
the contradiction between efforts to provide service through research and to 
build theory through research. Theory-building research must be, Kuhn 
argued, driven by the logic of the investigation and the availability of 
appropriate tools, not by the press of social needs. (One must be careful, 
as Kuhn was, not to overdraw this distinction between research driven by 
intellectual interests and that justified as meeting societal interests. Much 
basic research is motivated by concern for alleviating human suffering, for 
example, from cancer.) In social education, the paradox is that the research 
has met neither the .service needs or the demands of scientific knowledge. 



*I will not define "theory" extensively here., as I think our common 
understanding of what is meant by "theory", that is, nomothetic, scientific 
theory, as commonly used from the empirical perspective will suffice: that 
is, theories ". . . 'ideally tell us the necessary and sufficient 
conditions for a particular result 1 , allow the forecasting of outcomes 
'with a reasonable margin of error 1 , once parameters are specified, and 
include statements of the 'boundary conditions that limit [their] 
application 1 (Cronbach, 1975, p. 125; also see Larkins & McKinney, 1980, p. 
14)" (Shaver, 1982a, pp. 2-3). 

**For recent examples of discussions of the pract:ice-theory dilemma for 
researchers in other fields, see Howard (1985), the responses to his 
article, and Applebee (1986). 



It may be that there is insufficient scholarly research activity for 
either goal to be accomplished. Or, it may be that the empirical-scientific 
epistemological model is inappropriate for research in social education. 
That is, clearly, we lack the obvious progress from research which Kuhn 
(1970, p. 160-161) indicates is essential to science.* Also largely absent 
from the research in social education is at least one methodological 
characteristic of those fields that progress so obviously that they are 
labeled science, as Kuhn pointed out, with little regard for the niceties of 
definition— that is, a commitment to programs of research and to replication 
(see, e.g., Shaver & Norton, 1980). The question that follows is whether our 
scanty progress is due to lack of effort, faulty research strategy, or 
neither? 

Is the Empirical-Scientific Approach Appropriate? 

Is the empirical-scientific approach appropriate for building knowledge 
in social education? If that epistsmology were properly implemented in 
social education research, would more useful answers for practitioners or 
theory be the outcome? Is the reality of interest to us susceptible to 
"scientific progress r , as Kuhn (1979) used the term to refer to the natural 
sciences? 

In that regard, I continue to be impressed by the arguments put forth by 
Gergen (1973) and Cronbach (1975) that findings in the social sciences do not 
hold still long enough to be accumulated as knowledge or built into theory. 



*Kuhn (1970) make;., this point as a rather circular definition — that "the term 
'science 1 is reserved for fields that do progress in obvious ways" (p. 
160) — as an explanation for the lack of debate by those in the natural 
sciences as to whether their field is "really a science" as contrasted with 
the debates of that sort among "social scientists". 



That is, while human behavior is orderly and lawful, those who would 
formulate the laws of human behavior must take into account the continuous 
cultural changes, including those due to research, that condition human 
frames of reference and the resulting behavior. Those who try to formulate 
such laws will always be faced with the yet to be revealed effects of 
history. (Also see Campbell, 1986.) 

Cronbach (1975), going one step further, has pointed out that human 
behavior not only interacts with historical context, but with personological 
factors and ecological environments as well. The almost unlimited number of 
potential interactions to be ferreted out, Cronbach claims, makes the pinning 
down of the factors by wnich to explain behavior well nigh impossible (and 
perhaps accounts for many of the inconsistent, even contradictory, findings 
in educational research). (See Campbell, 1986, too.) The compelling and 
perplexing question raised by Cronbach's position is, what moderating 
variables ought to be taken into account in interpreting one's findings, but 
were not investigated, and perhaps could not be investigated because they are 
part of a context of application which has not yet occurred? 

The purpose of research from the Gergen-Cronbach perspective must be not 
to build nomothetic theory, but to sensitize people to the range of factors 
that may influence behavior in specific situations and to the relative 
importance of these factors (Gergen, 1973, p. 317); or, as Cronbach (1975) 
put it, "to develop . . . concepts which will help people use their heads" 
(p. 126) — a position harmonious with those who, so far without much fruition, 
aim to improve educational practice through research. It also sounds 
strangely like the purpose of critical research as enumerated by Popkewitz 
(1986, p. 2C} 3 



Note again that Gergen and Cronbach do not reject the premise that human 
events are lawful, but only quarrel with the idea that the assumed 
regularities can be somehow formulated into nomothetic scientific theory. 
But perhaps even the assumption of the lawfulness and regularity of human 
behavior needs to be challenged in pondering whether empirical-scientific 
epistemology is likely to lead to cumulative knowledge and theory in social 
education. Charles Perrow (1981), a sociologist who spent a great deal of 
his career "discovering 11 and writing about social organization, has, in a 
provocative essay, attacked that "conventional wisdom" of the social 
sciences. Perrow contends that, contrary to the scientific-empirical 
assumption of lawfulness and regularity in human behavior, there is 
considerable nonrationality, natural disorder, and unpredictability to human 
life. This claim, of course, runs counter to the social science goal of 
building rational designs to explain human behavior and thus eliminate — 
through what Perrow refers to as "convenient fictions" (p. 3) — disorder and 
unpredictability. 

Perrow argues that our drive to give "sensible accounts" of human 
phenomena — that is, to build theory — leads social scientists to disregard 
"happenstance, accidents, mysteries, illogicalities, and above all, fate" (p. 
4) as unforseeable determiners of human lives and social events. That was 
not the case, he points out, with the ancients, who tried to "make sense out 
of things", but also accepted the limits of rationality in discerning 
patterns of personal and social life, to the extent that such patterns 
exist. As Perrow (1981) summed up: "Count no life happy, the chorus repeats 
in the Greek tragedies, until it is over; one can never know what the 
unpredictable gods have in store" (p. 4). 
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Taken together, the critiques by Gergen, Cronbach, and Perrow suggest a 
different representation of social education than that of discoverable, 
predictable regularities. Our failure to accumulate knowledge may not be due 
to our imperfect implementation of scientific methodology, but to the 
inadequacy of that epistemology to the reality which we are trying to 
comprehend. Nonquanti tative methodologies, such as ethnography, will, I 
suspect, turn out not to be panaceas either, as their users strive to develop 
generalized propositions about social education. The data produced by 
absorption in the life of a few classrooms or through a critical perspective 
(see Stanley, 1986, p. 89) are likely to prove as inadequate a basis for 
generalization as traditional historians have, appropriately, perceived their 
data to be (Perrow, 1981). 

Meta-analysis 

Where does that new great hope, meta-analysis, fit into this picture? I 
suspect that the answer is fairly obvious. Let me note that I am in the 
midst of a federally- funded meta-analysis of some 273 reports, with over 700 
effect sizes, of research to modify attitudes toward disabled persons. I had 
methodological as well as substantive reasons for wanting to do such a review 
of literature. I am interested in the effects of negative attitudes toward 
disabled persons and, therefore, in how those attitudes might be modified. 
But I also wanted the opportunity to conduct a meta-analysis so that I could 
come to a better understanding of the method, as I was an early critic 
(Shaver, 1979) of Glass' (1976) concept of meta-analysis. I argued then that 
it was an inadequate quantitative answer to the problems of insufficient 
knowledge accumulation created by overreliance on inferential statistics and 



inattention to the basic scientific strategy of replication. I have not 
changed my mind. 

All that I have said above about the accumulation of knowledge through 
empirical-scientific methodology applies to meta-analysis. I am becoming 
more convinced through my own work that we stand to gain little from 
quantitative syntheses of primary research studies that are poorly designed 
and unrelated to one another. Certainly, attempts at quantitative synthesis 
are not akin to replication in the scientific sense, as has been implied by 
some authors (Jackson, 1980, p. 445; Banger t-Downs , 1986, p. 390). 
Attempting to infer explanatory patterns retrospectively is an entirely 
different logic from that of the prospective process of replication to 
determine whether findings will be reproduced and within what limits (Piatt, 
1964). 

A dogma among the rapidly emerging set of meta-analysis methodology 
specialists (e.g., Glass, 1976; Glass, McGaw, & Smith, 1981; Jackson, 1980; 
Rosenthal, 1984; Hedges & Olkin, 1985; Hunter, Schmidt, & Jackson, 1982; 
Wolf, 1986; Slavin, 1986) is that the steps in a quantitative review of 
literature are parallel to those in conducting a primary investigation. In 
that sense, those who do meta-analyses face — or should face — many of the same 
problems as those carrying out primary research studies. For example, the 
sampling problem which has plagued primary researchers (e.g., Shaver & 
Norton, 1980) is handled to some extent by the tactic, which we have adopted 
in doing our integrative review, of obtaining as complete a set as possible 
of published and unpublished research on the topic — that is, by attending to 
an accessible population rather than to a sample. 

The most troublesome aspect of a meta-analysis to me has its direct 
parallel in primary research, that is, the adequacy of assessment devices. 



ERLC 



9 10 



There is, in my experience, little difficulty in attaining reliability in the 
coding of study characteristics, but serious questions of validity are raised 
by the difficulties in anticipating and capturing the nuances of various 
primary studies conducted under differing conditions with varying samples and 
with the research reported with widely divergent degrees of detail and 
insightfulness into the elements of design and execution that might have 
moderated effects. The validation of meta-analysis coding instruments has 
hardly been treated in the literature, certainly not adequately. I urge you 
to scrutinize carefully the Instrumentation section of any meta-analysis 
report before deciding whether to rely on the findings and conclusions. 

Also bothersome is a strong push to once again rely on inferential 
statistics in analyzing data, ignoring both the inadequacies of number- 
dependent indicators of significance and the lack of relevance of statistical 
inference when analyzing data from populations rather than samples (although 
there is the sticky connundrum of whether to consider the primary studies 
being reviewed or the populations and settings in which the findings might be 
applied as the appropriate target in drawing conclusions—see, e.g., Jackson, 
1980, p. 453). Once again, as Wilson and Rachman (1983) have pointed out, 
there is the "danger that [the use of] sophisticated statistical techniques 
[will] serve [to] obscure damaging flaws in the evidence" (p. 55). And there 
is the ever present danger of fallacious reasoning based on misplaced 
precision as study characteristics are coded as numbers and effect sizes are 
computed to two or more decimal places. 

Participation in the coding of our 273 studies has given me e feeling 
for the myriad aspects of design and context that can affect results. 
Hunter, Schmidt, and Jackson (1982, pp. 32, 139) have pointed out that, in 
their experience, the greatest percentage of variability in outcomes among 
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primary studies can be accounted for by contextual factors and research 
artifacts, including sampling error, leaving little to be explained in terms 
of the treatment variable. I suspect our analysis will support that 
conclusion. 

If a meta-analysist wants straightforward answers, a simple coding 
system should be used so that the many possible interactions between 
treatments and contextual factors and research artifacts do not have to be 
taken into account. The analogue, of course, would be the use of one-way 
analysis of variance, rather than complex designs, so that one does not have 
to confront the difficulties in interpretation imposed by statistically 
significant interaction effects, much less consider all of the potential 
interactions which have not been investigated but which likely confound Lhe 
interpretation of results. 

I believe it was Chaim Potok who pointed out that for most people the 
need to know is much greater than the need to know what is true. I suggest 
that his insight explains much of the interpretation of research in 
education, perhaps applying especially to the answers being sought via meta- 
analysis. 

Summing Up 

The tone of this presentation is not meant to be negative, but hopefully 
provocative in a dialectic sense. I have tried to raise some issues related 
to the web of circumstances and epistemological difficulties that appear to 
bind and confound our efforts at research in social education from the 
empirical-scientific perspective. These issues include: What is the place 
of scholarship in social education? whafc are the goals of that scholarship 
(assuming that they are more than having publications for the sake of tenure 
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and promotion)? And, in particular, can research, especially empirical- 
scientific research, but ethnography and critical analysis as well, be 
carried out in such a way as to meet those goals? 

Calling attention to the unpredictability of human behavior, even to the 
role of fate, is not intended to be an argument against our efforts to be 
rational. It is meant rather to call attention to the bounds of rationality, 
to use Simon's (1983) terminology, and to the limits of educational research 
that is based on untenable assumptions about human behavior, in order that we 
might develop more fruitful perspectives on the role of research. Our past 
research endeavors in social education have not been particularly productive 
of either scientific theory or knowledge to inform or sensitize decision- 
makers. The first does not seem likely. How to better produce the latter is 
an important agenda item for social education researchers. (Parts of this 
section are taken from Shaver, 1982a, b. ) 
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